@th0ma7 I think I need your help.
I wanted to update SABnzbd last week, but I ran into the problem this PR fixes and also fixed it, but ran into the next problem.
It seems the building of the Python modules maturin (orjson) and pycparser suddenly broke. Strangely, not all archs fail at the same spot.
My build:
https://github.com/Safihre/spksrc/actions/runs/10557521724
6.2.4 fails at pycparser: https://github.com/Safihre/spksrc/actions/runs/10557521724/job/29245379175
7.1 fails at maturin: https://github.com/Safihre/spksrc/actions/runs/10557521724/job/29245380034

As you can see I tried to add the semantic-version package to the python-cross package, but it doesn't seem to help. I also tried putting it as part of python-native in another build, but that also didn't work.

Thoughts? :)

@Safihre There is this new issue that came up with inline assembly (tukaani-project/xz#121) for which there is already a fix available at tukaani-project/xz#136 that I ported into this PR. This will most likely be included in the next 5.6.3 release of ncurses.

That should already help so feel free to test this up.

@Safihre having any PR opened so I can have a look at your changes for sabnzbd?

I'm unable to trigger the python build errors using by Debian 12 container, merging #6183 and rebasing.

@hgy59 and @Safihre I am unable to reproduce the issue locally. Further I'm unable to download the logs, it always fails before ending the download. Although, while trying to look at raw logs online I was able to notice (before my browser crashed) something related to cffi==1.15.1 requested for install in the crossenv vs a cffi==1.17.0 needed by cryptography. So shot at the dark at updating them both, which built ok locally...

Also, in order to limit build time I've killed-off ffmpeg4 as I have no intention at maintaining it any longer.

Wait-and-see ... 🤷

@th0ma7 what about a dedicated PR with the xz fix?
This would fix the build of some other packages and decouple from this PR with the neverending python dependency drama...

Another dedicated PR could update ncurses if mandatory...

Excellent idea, I'll divide and conquer... And hopefully solve the python drama indeed.

This would fix the build of some other packages and decouple from this PR with the neverending python dependency drama...

Another dedicated PR could update ncurses if mandatory...

That's what I did with #6208, #6209 and #6210

Although, to be confirmed, but I'm guessing they may end-up being co-dependent to xz download issue, which ended-up heading to this rabbit hole.

Although, to be confirmed, but I'm guessing they may end-up being co-dependent to xz download issue, which ended-up heading to this rabbit hole.

Yes, xz must be merged before the others

And ncurses (AFAICR) triggers too many packages to complete within 5 hrs....

I tried to build python311 on this branch (with all wheels: WHEELS_TEST_ALL = 1)

• The build fails and aborts when building pydantic wheel (ModuleNotFoundError: No module named 'typing_extensions')

• This can be fixed by adding typing_extension ('typing-extensions >=4.6.0,!=4.7.0') to the cross env, but you didn't include the enhancement for ADDITIONAL_CROSSENV_WHEELS of Python 3.11.8 update #6040

• Then the the build aborts when building numpy wheel. This is really a mess, it says that setuptools must be < 60.0 for numpy.distutils to work - and will not be supported for python >= 3.12 (https://numpy.org/devdocs/reference/distutils_status_migration.html)
  and indeed, to build numpy wheel (any version 1.25/1.24/1.22) we must use setuptools==59.2.0

• future versions of numpy (1.26+) need meson-python instead of setuptools (as you commented in requirements-crossenv-numpy.txt)
  meson-python can already be used for numpy 1.25, but needs some (prepared and documented) patches of pyproject.toml

• general wheel build enhancement:
  since more and more wheels use pyproject.toml (instead of deprecated setup.py),
  we should install the "build-system requirements defined in pyproject.toml" into the cross env
  may be with such an enhancement, the ADDITIONAL_CROSSENV_WHEELS will be obsolete

• what was the reason to define "ADDITIONAL_WHEEL_BUILD_ARGS=--no-build-isolation" ?
  AFAIS without build isolation, we have to ensure all build dependencies are installed,
  otherwise pip would take care of this...
  probably the reason was to reuse the manually created cross env, but this is in contradiction to wheel specific build dependencies...
  "--no-build-isolation" was introduced long time ago (first seen in cross/python38/Makefile, not present in former cross/python3/Makefile)

@th0ma7 can you run a build with WHEELS_TEST_ALL = 1?

I did this with a former version of this PR (with python311 v3.11.5) and it worked when numpy was downgraded to 1.24.4 in requirements-crossenv-numpy.txt (same version as in requirements-crossenv-numpy-gcc4.txt).

additionally I downgraded setuptools in crossenv to v68.1.2, but I don't know whether it is requiered.

My plan of action is to merge this pr as is to have a python update provided asap.

Next is creating a new pr to have latest vs legacy wheel build environments or something similar. From my previous testing, up to the crossenv version used as an impact, in particular with older wheels.

Also, part of the build issues found is the inability to pass toolchain files for cmake and meson based wheels. I believe i may have a solution for that.

Edit: also, i intend to move all wheels currently into python31* spk under its own spk for more in-depth testing and easier update releases.

I tried to build python311 on this branch (with all wheels: WHEELS_TEST_ALL = 1)

• The build fails and aborts when building pydantic wheel (ModuleNotFoundError: No module named 'typing_extensions')

• This can be fixed by adding typing_extension ('typing-extensions >=4.6.0,!=4.7.0') to the cross env, but you didn't include the enhancement for ADDITIONAL_CROSSENV_WHEELS of Python 3.11.8 update #6040

• Then the the build aborts when building numpy wheel. This is really a mess, it says that setuptools must be < 60.0 for numpy.distutils to work - and will not be supported for python >= 3.12 (https://numpy.org/devdocs/reference/distutils_status_migration.html)
  and indeed, to build numpy wheel (any version 1.25/1.24/1.22) we must use setuptools==59.2.0

• future versions of numpy (1.26+) need meson-python instead of setuptools (as you commented in requirements-crossenv-numpy.txt)
  meson-python can already be used for numpy 1.25, but needs some (prepared and documented) patches of pyproject.toml

• general wheel build enhancement:
  since more and more wheels use pyproject.toml (instead of deprecated setup.py),
  we should install the "build-system requirements defined in pyproject.toml" into the cross env
  may be with such an enhancement, the ADDITIONAL_CROSSENV_WHEELS will be obsolete

• what was the reason to define "ADDITIONAL_WHEEL_BUILD_ARGS=--no-build-isolation" ?
  AFAIS without build isolation, we have to ensure all build dependencies are installed,
  otherwise pip would take care of this...
  probably the reason was to reuse the manually created cross env, but this is in contradiction to wheel specific build dependencies...
  "--no-build-isolation" was introduced long time ago (first seen in cross/python38/Makefile, not present in former cross/python3/Makefile)

I had not noticed your previous msg. I'll look into this while working on the subsequent pr.

@th0ma7 with the changes I just committed all wheels are built successfully in my local environment (except qoriq and ARMv7L that have an issue with rust that is probably only in my environment).
I tried the same changes in python310 but this fails with other wheels for other reason (cryptography) so I left python310 as is.

Thnx for the commit, it does solves the immediate issue afaics. Although missing are:

• github action no longer kick-off dependent python based spk anymore as no longer directly call from depend, thus not sure if and how packages do build
• downgrading some like setuptools and other fixes the immediate issues, but clearly something else will be needed before long to manage those as we can't stay with using legacy cython version forever for instance
• and pip options changed over time whereas some old arguments meant to solve earlier issues no longer works or are set to be orphaned.
• cmake and meson wheel builds not yet addressed

All to say, maybe we still can get along with this round but clock is ticking before we need to make more fundamental changes, which i thought were needed now (with your commits, maybe for the next round).

All to say, maybe we still can get along with this round but clock is ticking before we need to make more fundamental changes, which i thought were needed now (with your commits, maybe for the next round).

IMHO we should avoid fundamental changes in python311 and earlier. But for python312 we definitifly need a redesign of the cross wheel building process.

Thanks for all your hard work both!